Contributing Intel Tofino compiler backend to p4c #4964

hanw · 2024-10-17T02:36:49Z

This pull request introduces the compiler backend for Intel Tofino 1 and Tofino 2 programmable switch ASICs, licensed under the Apache License 2.0. Due to the large number of files, this PR has been divided into a series of commits, organized primarily by directory structure.

To build the backend, use the -DENABLE_TOFINO=ON flag during bootstrapping. This flag is disabled by default.

Please note that this PR contains only the source code for the Tofino backend and does not include additional regression tests. A separate PR will be submitted to provide a set of PTF and STF tests for this backend.

A big thank you to all the team members who contributed to this compiler -- you know who you are!

-- Glen, Prathima, Han

fruffy

Exciting!

I would consider using something like https://github.com/p4lang/p4c/blob/main/backends/p4tools/cmake/common.cmake#L16 to fix the Z3 issues. At least it has worked well in the tools back end. We can put that script into top-level cmake folder as Z3.cmake.

fruffy · 2024-10-17T12:11:48Z

CMakeLists.txt

@@ -35,6 +35,7 @@ set (CMAKE_USE_RELATIVE_PATHS 1)

 OPTION (ENABLE_DOCS "Build the documentation" OFF)
 OPTION (ENABLE_GTESTS "Enable building and running GTest unit tests" ON)
+OPTION (ENABLE_TOFINO "Build the TOFINO backend (required for the full test suite)" ON)


I would disable this by default and only build Tofino on the Ubuntu 22.04 Github actions.

Another thing is that this backend is really... large :) So not all breaking changes might be quite non-trivial :)

Another thing is that this backend is really... large :) So not all breaking changes might be quite non-trivial :)

This might actually be a good thing. bf-p4c pushes the limits of what you can do as a back end. So if we introduce breaking changes it can act as a canary.

Yes, it has both pros and cons. I am talking mostly about the amount of possible changes required if we'd introduce, say, some IR change or something like this :)

fruffy

Initial round of comments.

fruffy · 2024-10-21T12:01:31Z

CMakeLists.txt

@@ -374,6 +375,7 @@ include_directories (
  ${P4C_SOURCE_DIR}
  ${P4C_BINARY_DIR}
  ${P4C_SOURCE_DIR}/extensions
+  ${P4C_SOURCE_DIR}/backends/tofino  ## FIXME


I am guessing this refers to fixing the include paths?

Yes, this should not be needed -- but I guess just removing it currently breaks things? If it is needed, it should probably be in backends/tofino/CMakeLists.txt, but I'm unsure how cmake nesting works in general.

tools/ci-build.sh

fruffy · 2024-10-21T12:08:08Z

tools/driver/CMakeLists.txt

-add_test(NAME driver_inputs_test_2 COMMAND ${P4C_SOURCE_DIR}/tools/driver/test_scripts/driver_inputs_test_2)
-add_test(NAME driver_inputs_test_3 COMMAND ${P4C_SOURCE_DIR}/tools/driver/test_scripts/driver_inputs_test_3)
-add_test(NAME driver_inputs_test_4 COMMAND ${P4C_SOURCE_DIR}/tools/driver/test_scripts/driver_inputs_test_4)
+### Disabled to unblock CI for Tofino backend. Will reenable after I figure out the reason of the test failure.


Looks like the PR is blocked on this? What are the failures?

Not clear, probably because the script assumes the top level project name is p4c, instead of bf-p4c? I cannot reproduce the error locally.

Hold on we should fix this first before merging.

Can you give it a try to reproduce the error in your local environment/

When you remove the comments, what is the CI error? Will try to build bf-p4c soon, but not sure when.

In the bf-p4c repo, this is the error:

FATAL ERROR: could not find the P4 compiler driver. Searched in the following locations... ///>>>./p4c<<</// ///>>>../p4c<<</// ///>>>../../p4c<<</// ///>>>../p4c/p4c<<</// ///>>>../../p4c/p4c<<</// ///>>>/home/runner/work/bf-p4c/bf-p4c<<</// ///>>>/home/runner/work/bf-p4c/bf-p4c/p4c<<</// ///>>>/home/runner/work/bf-p4c/bf-p4c/p4c/p4c<<</// ///>>>/home/runner/work/bf-p4c<<</// ///>>>/home/runner/work/bf-p4c/p4c<<</// ///>>>/home/runner/work/bf-p4c/p4c/p4c<<</// ///>>>/home/runner/work/_temp//p4c<<</// ///>>>/home/runner/work/_temp//p4c/p4c<<</// Unable to find the driver of the P4 compiler. Aborting the test ''/home/runner/work/bf-p4c/bf-p4c/tools/driver/test_scripts/driver_inputs_test_3___three_bad_pathnames.bash'' with a non-zero exit code. This test failed.

backends/tofino/ubuntu20.04.setup

backends/tofino/cmake/FindZ3.cmake

backends/tofino/CMakeLists.txt

fruffy · 2024-10-21T12:24:06Z

backends/tofino/CMakeLists.txt

+  enable_testing()
+endif()
+
+set(BOOST_MIN_VERSION "1.58.0")


How much boost is left in the compiler? We got rid of a lot of it in the open-source compiler.

there are still some, phv depends on boost::graph.

I am seeing use of:

boost/range / boost::irange

boost/algorithms/string (use abseil instead)

boost/iostreams

boost/none / optional (certainly should be removed in favour of standard ones)

boost/uuid

boost/functional (mostly boost::hash, replace with Util::Hash)

boost/iterator

boost/format

boost/property_map

boost/bind (replace with lambda's and C++17 features)

boost/operators

boost/make_unique

Plus some code to support legacy pre-C++17 things here and there.

Okay, most of those we should be able to clean up.

#4663 could actually help with identifying extra boost stuff, the build should break for includes that we did not declare. Will rebase after this PR is merged.

backends/tofino/CMakeLists.txt

asl · 2024-10-22T03:33:16Z

Given the size of the backend – who is supposed to support it after the merge? I am worrying that any project-wide changes will become very time-consuming given the size of the backend. Maybe it should live in a separate repo and rebase from time to time to p4c mainline?

jafingerhut · 2024-10-22T04:19:15Z

Given the size of the backend – who is supposed to support it after the merge? I am worrying that any project-wide changes will become very time-consuming given the size of the backend. Maybe it should live in a separate repo and rebase from time to time to p4c mainline?

@grg @hanw Have you had any discussions about this topic? I thought that perhaps Glen had mentioned an approach like this at one point, and while there is a risk of the Tofino back end being left behind for long stretches of time until someone catches it up, it does have some advantages.

ChrisDodd

This is too large to properly review, but as a self contained backend, the main concern is how well it integrates with the rest of the code.

My recollection is that including this backend slows down the build-time A LOT, so that may be an issue.

I have a little bit of concern about the license, as it appears to not be Apache licensed, but IANAL, so I can only suggest getting the opinion of someone who is a copyright lawyer.

ChrisDodd · 2024-10-22T04:30:53Z

CMakeLists.txt

@@ -374,6 +375,7 @@ include_directories (
  ${P4C_SOURCE_DIR}
  ${P4C_BINARY_DIR}
  ${P4C_SOURCE_DIR}/extensions
+  ${P4C_SOURCE_DIR}/backends/tofino  ## FIXME


Yes, this should not be needed -- but I guess just removing it currently breaks things? If it is needed, it should probably be in backends/tofino/CMakeLists.txt, but I'm unsure how cmake nesting works in general.

ChrisDodd · 2024-10-22T04:39:43Z

backends/tofino/CMakeLists.txt

+  COMMENT "Building ctags")
+
+add_custom_target(cpplint-all
+  DEPENDS cpplint-asm cpplint


Since bfasm isn't present in this code, there probably should not be dependencies on it here?

ChrisDodd · 2024-10-22T04:49:39Z

backends/tofino/bf-p4c/.gdbinit

+set print object
+set unwindonsignal on
+set unwind-on-terminating-exception on
+set python print-stack full


This should probably be merged with the top-level .gdbinit file -- it shares a lot in common with that in any case

kfcripps · 2024-10-22T11:25:24Z

@grg @hanw Have you had any discussions about this topic? I thought that perhaps Glen had mentioned an approach like this at one point, and while there is a risk of the Tofino back end being left behind for long stretches of time until someone catches it up, it does have some advantages.

We may consider either:

Merge Tofino into a different branch instead of main, and only periodically update this branch, or
Include all Tofino-related CI checks under an optional PR label, and tag each commit for which this label's job(s) are passing, or
Add Tofino to a different fork as @asl suggested (this is very similar to (1) in terms of maintenance)

fruffy · 2024-10-22T13:44:48Z

Seems like my comment ended up in the ether somewhere, reposting:

Given the size of the backend – who is supposed to support it after the merge? I am worrying that any project-wide changes will become very time-consuming given the size of the backend. Maybe it should live in a separate repo and rebase from time to time to p4c mainline?

We discussed this and the best option is to have the Tofino compiler as part of the open-source P4C to avoid bit-rotting. However, in order to not encumber the rest of the compiler too much this back end will be part of a separate PR that is optional.

There is an advantage to this approach: the front end of the compiler has largely been designed for the Tofino back end. Now that both are integrated with each other we can start to be more aggressive in refactorings or optimizations because we immediately see the effects on the Tofino back end. I expect things to improve quickly.

We can also see how this works out in practice over the next couple of weeks.

@grg @hanw Have you had any discussions about this topic? I thought that perhaps Glen had mentioned an approach like this at one point, and while there is a risk of the Tofino back end being left behind for long stretches of time until someone catches it up, it does have some advantages.

We may consider either:
1. Merge Tofino into a different branch instead of `main`, and only periodically update this branch, or

2. Include all Tofino-related CI checks under an optional PR label, and tag each commit for which this label's job(s) are passing, or

3. Add Tofino to a different fork as @asl suggested (this is very similar to (1) in terms of maintenance)

We will probably go with option 2, but have it run similar to the Fedora tests (on every commit, but not required for merging).

asl · 2024-10-22T16:19:19Z

We discussed this and the best option is to have the Tofino compiler as part of the open-source P4C to avoid bit-rotting. However, in order to not encumber the rest of the compiler too much this back end will be part of a separate PR that is optional.

Well, my feeling is that it puts very huge burden on not-so-large community and effectively prevents many project-wide changes. Certainly, it could be optional and not required for merging. So, does this mean, it will just become broken at any breaking change, is this the intention? Bit-rotting is unavoidable if there is noone to support it.

Now that both are integrated with each other we can start to be more aggressive in refactorings or optimizations because we immediately see the effects on the Tofino back end. I expect things to improve quickly.

Let me drop a small example. Let's say I need to do some IR-related refactoring that requires changes everywhere. If I need to spend, say, 8 hours to do this for frontend + midend + existing backends and spend another, say, 40 hours just for Tofino, will I be willing to make such change? Highly unlikely. Or I will just leave Tofino on the side and expect someone else to make this port?

hanw · 2024-10-22T16:38:05Z

We discussed this and the best option is to have the Tofino compiler as part of the open-source P4C to avoid bit-rotting. However, in order to not encumber the rest of the compiler too much this back end will be part of a separate PR that is optional.

Well, my feeling is that it puts very huge burden on not-so-large community and effectively prevents many project-wide changes. Certainly, it could be optional and not required for merging. So, does this mean, it will just become broken at any breaking change, is this the intention? Bit-rotting is unavoidable if there is noone to support it.

Now that both are integrated with each other we can start to be more aggressive in refactorings or optimizations because we immediately see the effects on the Tofino back end. I expect things to improve quickly.

Let me drop a small example. Let's say I need to do some IR-related refactoring that requires changes everywhere. If I need to spend, say, 8 hours to do this for frontend + midend + existing backends and spend another, say, 40 hours just for Tofino, will I be willing to make such change? Highly unlikely. Or I will just leave Tofino on the side and expect someone else to make this port?

Intel will have engineers to maintain the code. The fact that it's open-sourced makes it easier for people that are not at Intel to contribute.

fruffy · 2024-10-22T17:41:51Z

backends/tofino/LICENSE

@@ -0,0 +1,10 @@
+Copyright 2013-2024 Intel Corporation.


@hanw @grg Is this a new license or is it just outdated?

It's outdated, will fix.

asl · 2024-10-22T17:55:49Z

Intel will have engineers to maintain the code. The fact that it's open-sourced makes it easier for people that are not at Intel to contribute.

Thanks for the clarification. So, am I reading the policy correctly?

The Tofino backend will be optional for new contributions
It may be broken in the meantime and Intel engineers will unbreak it

jafingerhut · 2024-10-22T18:33:55Z

Intel will have engineers to maintain the code. The fact that it's open-sourced makes it easier for people that are not at Intel to contribute.

Thanks for the clarification. So, am I reading the policy correctly?

The Tofino backend will be optional for new contributions

It may be broken in the meantime and Intel engineers will unbreak it

And if this is true, it would be good to always publish a short list of commit SHAs somewhere, for which the Tofino back end builds, and has been well-tested.

fruffy · 2024-10-22T18:55:17Z

Let me drop a small example. Let's say I need to do some IR-related refactoring that requires changes everywhere. If I need to spend, say, 8 hours to do this for frontend + midend + existing backends and spend another, say, 40 hours just for Tofino, will I be willing to make such change? Highly unlikely. Or I will just leave Tofino on the side and expect someone else to make this port?

I had to port breaking changes to bf-p4c in the past. I also currently have a couple of PRs open that will require fixing once the Tofino compiler is merged. Overall, the code base seems daunting but a lot of the code is fairly self-contained. I estimate things will only improve once we start to clean up the back end and move some of its parts into the core. There is a lot of redundancy that we can get rid of.

I'd say we evaluate how things are going for a couple of weeks and if we notice that the burden is to high we can still factor out the compiler. There is very little practical difference between a back end and an extension so it should not be a big change.

hanw · 2024-10-23T00:27:02Z

Intel will have engineers to maintain the code. The fact that it's open-sourced makes it easier for people that are not at Intel to contribute.

Thanks for the clarification. So, am I reading the policy correctly?

The Tofino backend will be optional for new contributions

It may be broken in the meantime and Intel engineers will unbreak it

This aligns with my understanding.

hanw · 2024-10-23T00:27:25Z

Let me drop a small example. Let's say I need to do some IR-related refactoring that requires changes everywhere. If I need to spend, say, 8 hours to do this for frontend + midend + existing backends and spend another, say, 40 hours just for Tofino, will I be willing to make such change? Highly unlikely. Or I will just leave Tofino on the side and expect someone else to make this port?

I had to port breaking changes to bf-p4c in the past. I also currently have a couple of PRs open that will require fixing once the Tofino compiler is merged. Overall, the code base seems daunting but a lot of the code is fairly self-contained. I estimate things will only improve once we start to clean up the back end and move some of its parts into the core. There is a lot of redundancy that we can get rid of.

I'd say we evaluate how things are going for a couple of weeks and if we notice that the burden is to high we can still factor out the compiler. There is very little practical difference between a back end and an extension so it should not be a big change.

I agree with this approach.

fruffy · 2024-10-23T12:23:25Z

.github/workflows/ci-test-debian.yml

@@ -21,6 +21,7 @@ jobs:
      CTEST_PARALLEL_LEVEL: 4
      IMAGE_TYPE: test
      BUILD_GENERATOR: Ninja
+      ENABLE_TOFINO: ON


I would create a new job test-ubuntu22-tofino` which is a copy of this job and enable Tofino there. We can also disable all other back ends to save compilation time.

fruffy · 2024-10-23T15:41:29Z

tools/driver/CMakeLists.txt

-add_test(NAME driver_inputs_test_2 COMMAND ${P4C_SOURCE_DIR}/tools/driver/test_scripts/driver_inputs_test_2)
-add_test(NAME driver_inputs_test_3 COMMAND ${P4C_SOURCE_DIR}/tools/driver/test_scripts/driver_inputs_test_3)
-add_test(NAME driver_inputs_test_4 COMMAND ${P4C_SOURCE_DIR}/tools/driver/test_scripts/driver_inputs_test_4)
+### Disabled to unblock CI for Tofino backend. Will reenable after I figure out the reason of the test failure.


Hold on we should fix this first before merging.

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

- fix ci-build.sh - add pyinstaller to requirement.txt - link ${P4C_LIB_DEPS} to bf_gtest_support target - link ${P4C_LIB_DEPS} to tofinobackend target Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

- clang-format - black format - isort errors - ifdefs name fixes Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

- remove duplicated bmv2/psa.p4 and dpdk/psa.p4 - remove calling bfas in p4c-driver - disable a few driver tests Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

hanw · 2024-10-23T20:56:00Z

It would be great if some trivial dependencies from boost are cleaned before merge (e.g. boost::hash or things that are in C++17)

This PR is large enough. Let's visit boost libraries in a follow-up PR. I also need to add a regression suite before further editing the source code.

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

fruffy · 2024-10-23T21:45:07Z

Let's wait until Friday in case someone else has concerns. If there are no further comments at that point we can merge this PR then.

vlstill · 2024-10-30T07:50:33Z

@hanw, any chance the Tofino Testgen target will be also contributed?

fruffy · 2024-10-30T12:19:16Z

@hanw, any chance the Tofino Testgen target will be also contributed?

@pkotikal

fruffy added the tofino Topics related to the Tofino switch and back end. label Oct 17, 2024

fruffy reviewed Oct 17, 2024

View reviewed changes

hanw force-pushed the tofino-backend branch from 25909fb to cba46af Compare October 21, 2024 00:51

hanw requested a review from grg October 21, 2024 01:55

fruffy requested review from vlstill, asl and ChrisDodd October 21, 2024 12:00

fruffy reviewed Oct 21, 2024

View reviewed changes

fruffy requested a review from kfcripps October 21, 2024 15:07

fruffy mentioned this pull request Oct 22, 2024

Add abseil string helpers #4971

Merged

ChrisDodd approved these changes Oct 22, 2024

View reviewed changes

fruffy reviewed Oct 22, 2024

View reviewed changes

hanw force-pushed the tofino-backend branch from de088c3 to 2ec8e73 Compare October 23, 2024 00:24

fruffy reviewed Oct 23, 2024

View reviewed changes

hanw force-pushed the tofino-backend branch from 2ec8e73 to d82ec3d Compare October 23, 2024 14:46

hanw enabled auto-merge October 23, 2024 15:28

hanw added this pull request to the merge queue Oct 23, 2024

fruffy removed this pull request from the merge queue due to a manual request Oct 23, 2024

fruffy requested changes Oct 23, 2024

View reviewed changes

hanw and others added 15 commits October 23, 2024 13:47

Add files under backends/tofino/compiler-interfaces

7f9ed98

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

Add files under backends/tofino/scripts for misc scripts

8084bc1

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

Changes to other backend to fix compilation error

9f48981

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

Changes to top level CMakeLists.txt to add tofino backend

2c5b518

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

fix compilation issues

a93e1d3

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

switch ENABLE_TOFINO to OFF by default

e53889d

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

try 20.04 build

73ccba1

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

add FindZ3.cmake for bf-p4c

a64e6b1

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

Addressing review comments

439d803

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

Update licenses to Apache 2.0

6663b2d

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

Create new test runner for tofino backend

5e3ba6c

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

reenable driver tests

cd439da

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

hanw force-pushed the tofino-backend branch from 8e499db to a19b24d Compare October 23, 2024 20:53

address review comments

0e0c88e

Co-authored-by: Glen Gibb <[email protected]> Co-authored-by: Prathima Kotikalapudi <[email protected]> Co-authored-by: Han Wang <[email protected]> Signed-off-by: Han Wang <[email protected]>

hanw force-pushed the tofino-backend branch from a19b24d to 0e0c88e Compare October 23, 2024 20:59

hanw requested a review from fruffy October 23, 2024 21:17

fruffy enabled auto-merge October 25, 2024 21:28

fruffy approved these changes Oct 25, 2024

View reviewed changes

fruffy added this pull request to the merge queue Oct 25, 2024

Merged via the queue into main with commit 2605767 Oct 25, 2024
19 checks passed

fruffy deleted the tofino-backend branch October 25, 2024 22:23

fruffy mentioned this pull request Oct 31, 2024

WEDGE Switch installation #4988

Open

Contributing Intel Tofino compiler backend to p4c #4964

Contributing Intel Tofino compiler backend to p4c #4964

Conversation

hanw commented Oct 17, 2024 • edited Loading

fruffy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fruffy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fruffy Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asl commented Oct 22, 2024

jafingerhut commented Oct 22, 2024

ChrisDodd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfcripps commented Oct 22, 2024

fruffy commented Oct 22, 2024

asl commented Oct 22, 2024 • edited Loading

hanw commented Oct 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asl commented Oct 22, 2024

jafingerhut commented Oct 22, 2024

fruffy commented Oct 22, 2024

hanw commented Oct 23, 2024

hanw commented Oct 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanw commented Oct 23, 2024

fruffy commented Oct 23, 2024

vlstill commented Oct 30, 2024

fruffy commented Oct 30, 2024

hanw commented Oct 17, 2024 •

edited

Loading

fruffy Oct 23, 2024 •

edited

Loading

asl commented Oct 22, 2024 •

edited

Loading